Apparatus for identification of an object queue, method and computer program

ABSTRACT

In daily life, people are often forced to join a queue in order, for example, to pay at a checkout or to be dealt with at an airport, etc. Because of the various forms of a queue, these are not usually recorded automatically, but are analyzed manually. For example, if a long queue is formed at a supermarket, as a result of which the predicted waiting time for the customers rises above a threshold value, this situation can be identified by the checkout personnel, and a further checkout can be opened. A device  1  is proposed for identification of a queue  2  of objects  10  in a monitoring area, having an interface  6  which can be connected to an image source  7,  with the interface  6  being designed to observe at least one monitoring image  3  of the monitoring area of the image source, wherein the monitoring image  3  shows a scene background of the monitoring area with possible objects  10,  having an evaluation device  5  which is designed to identify the queue  2  of the objects  10  in the at least one monitoring image, wherein the evaluation device  5  has an object detector module  8  which is designed to detect a plurality of objects  10  on the basis of the monitoring image  3,  wherein the plurality of the detected objects  10  forms the basis for identification of the queue  2  of the objects  10,  wherein the object detector module  8  is designed to identify the objects  8  in the monitoring image with the scene background and/or wherein the object detector module  8  has content-sensitive detectors  9  for detection of the objects  10.

BACKGROUND OF THE INVENTION

The invention relates to an apparatus for identifying a queue of objectsin a monitoring area, having an interface which can be connected to animage source, the interface being designed to accept at least onemonitoring image of the monitoring area from the image source, themonitoring image showing a scene background of the monitoring area withpossible objects, having an evaluation device which is designed toidentify the queue of objects in the at least one monitoring image, theevaluation device having an object detector module which is designed todetect a multiplicity of the objects on the basis of the monitoringimage, and the multiplicity of the detected objects forming the basisfor identifying the queue of objects. The invention also relates to acorresponding method and a computer program.

In daily life, people are often forced to join a waiting queue in orderto pay at a checkout, to be dealt with at the airport, etc., forexample. Waiting queues occur in many kinds of scenarios and are formedwhen the arrival rate of objects exceeds the processing rate. Thearrival rate of the objects is often unknown and varies with time. Thislikewise applies to the processing rate in many cases. Therefore, thelength of the waiting queue is an unknown quantity which cannot becalculated but must be determined in another manner.

With a fixed processing rate, such as in the case of a ride in anamusement park, it is possible to predefine the shape of the queue bymeans of a barrier, the likely waiting time then being able to bedetermined with the aid of empirical methods. The likely waiting timecan then be displayed to the person waiting using permanently fittedboards along the queue.

Another possibility for determining a waiting queue is provided if thewaiting area has clearly defined entrances and exits, with the resultthat the number of objects in the waiting area can be determined, forexample, using light barriers or turnstiles. However, no furtherstatement on the shape and number of waiting queues can then be madeinside the waiting area.

On account of the varying embodiments of a waiting queue, the latter isusually not detected in an automated manner but rather is usuallyanalyzed manually. If, for example, a long queue forms in a supermarket,with the result that the likely waiting time for the customers risesabove a threshold value, this circumstance can be ascertained by thecheckout staff and a further checkout can be opened.

In the documentary prior art, the document U.S. Pat. No. 5,581,625discloses a stereo camera system for counting objects in a queue. In thesystem, an item of depth information relating to the objects is acquiredby the stereo camera in order to count the objects.

In contrast, the document U.S. Pat. No. 5,953,055, which probably formsthe closest prior art, relates to a system and a method for detectingand analyzing queues. Said document proposes first of all subtracting alearnt background image from a monitoring image and classifying theremaining image pixels containing information as a queue.

SUMMARY OF THE INVENTION

Among other variations and embodiments, an apparatus for identifying aqueue having the features of claim 1, a method having the features ofclaim 11 and a computer program having the features of claim 13 areproposed.

Preferred or advantageous embodiments will be apparent from thesubclaims, the following description, and the accompanying figures.

An apparatus which is suitable and/or designed to identify a queue ofobjects in a monitoring area is proposed within the scope of theinvention. A queue, preferably a waiting queue, of objects is understoodas meaning an accumulation of these objects which is arranged in astraight line or a curved line, optionally additionally with a pluralityof branches, intermediate gaps, accumulations and/or agglomerations.Therefore, the term queue is preferably understood as meaning anyorganized or quasi organized arrangement of the objects queuing at oneor more destinations. The destination may be, for example, a checkout,an entrance, an exit or a functional area, for example a food counter, avending machine, etc.

The apparatus preferably comprises a data processing device, whichimplements the identification functions, and has at least one interfacewhich can be connected to an image source. The connection can be wiredor wireless. The image source is preferably in the form of a monitoringcamera, in particular a mono-image camera. This makes it possible tooperate the apparatus in real time. Alternatively or additionally, theimage source is in the form of an image memory which provides storedimages. At least one monitoring image of the monitoring area can bepassed to the apparatus via the interface, the monitoring image showinga scene background of the monitoring area with possible objects. Thescene background of the monitoring area is formed by the naturallyoccurring background, for example a street, shelves in a shop, etc.

In order to identify the queue in the at least one monitoring image, theapparatus has an evaluation device. The evaluation device includes anobject detector module which is designed to detect a multiplicity of theobjects on the basis of the monitoring image, the multiplicity of thedetected objects forming the basis for identifying the queue of objects.In one preferred embodiment of the invention, the objects are identifiedas separate objects or individual objects by the object detector module.

The the object detector module is designed to identify the objects inthe monitoring image with the scene background. Thus it is possible todispense with the background subtraction, which is used in the priorart. The background subtraction is a critical step in prior-art imageanalysis since a static background is often ideally used as the startingpoint for analysis. However, the background is usually only quasi staticin actuality. For example, the background image can be greatlyinfluenced by an automobile driving past, a change in light or otherdisturbances. In contrast, it is advantageous if this error-prone stepis dispensed with and the objects are immediately identified against thescene background in the monitoring image.

Alternatively or additionally, the object detector module hascontent-sensitive detectors for detecting the objects. Content-sensitivedetectors are understood as meaning detectors which find the objects byanalyzing the contents of the monitoring image. This procedure makes itpossible to look for and detect the objects against the scene backgroundin the monitoring image. However, it is also possible, purely inprinciple, to use these detectors in a preprocessed monitoring image inwhich the scene background has already been subtracted, as is known fromthe prior art.

The advantage of the invention can therefore be seen in the fact thatthe monitoring image can be analyzed in a different manner and thus in amanner which is less susceptible to faults with regard to the detectorsand/or the missing background subtraction.

In one preferred implementation of the invention, the content-sensitivedetector(s) is/are designed to use a multiplicity of comparison featuresand classification rules for the comparison features. The comparisonfeatures may be complex, with the result that the image or the shape ofan arm, a leg, etc. is used as the comparison feature, for example.Alternatively or additionally, the comparison features may also be basedon simple geometrical shapes, such as horizontal lines, vertical lines,round shapes for heads, etc. The literature discloses a multiplicity offeatures or feature systems for image processing which can accordinglybe selected for the objects. The content-sensitive detector(s) alsocomprise(s) classification rules which combine the results of thecomparison features in order to enable an overall statement “objectfound-not found”. The classification rules may comprise analytical ruleswhich are programmed in, for example. In this case, it is conceivablefor plausibility checks to be used as a classification rule, for examplethe fact that any person only has a maximum of two arms, with the resultthat an object having more than two arms is rejected. Alternatively oradditionally, the classification rules may also be learnt; amultiplicity of possibilities (for example boost methods, neuralnetworks, etc.) are also known in this respect.

In particular, the content-sensitive detector(s) is/are designed todetect and locate individual objects, preferably even in the case ofmutual or other concealment and/or covering.

Optionally, provision may additionally be made for the object detectormodule and/or the evaluation device to be designed to identify furtherevents in connection with queues, for example falling over, obstacles,queue-jumping, arguments, etc.

In one possible development of the invention, the evaluation device hasa modeling module which is designed to model the queue on the basis ofthe detected objects. Since the objects in the monitoring image were notonly detected but also preferably located, the modeling module can modelthe queue. In particular, the modeling module is designed to model anydesired curved and/or branched queue shapes. In principle, provision maybe made for additional information, for example an end point of thequeue, to be manually input as expert or a-priori knowledge.Alternatively or additionally, however, the modeling module operateswithout knowledge of the position or existence of the queue.

The modeling module particularly preferably has a multiplicity ofwaiting queue models which cover different versions of waiting queues.The modeling module is designed to place such a waiting queue model ontothe multiplicity of objects and/or to adapt the model to thedistribution of said objects. The waiting queue models can bedistinguished by the profile (for example curved or straight), thenumber or existence of branches, accumulations, etc.

In one particularly preferred embodiment of the invention, theevaluation device is designed, in terms of programming and/or circuitry,to identify and/or form the queue on the basis of a single monitoringimage. Advantages over the prior art, which requires at least twoimages, namely the monitoring image and the background image, toidentify the queue, are shown in this embodiment.

One development of the invention may provide for the evaluation deviceto have a movement analysis module which analyzes movement informationand provides it as additional information for identifying and/orverifying the queue. The evaluation device, in particular the modelingmodule, is designed to use the additional information to identify and/orverify the queue. The movement analysis module is designed, for example,to identify the direction of movement of the detected objects in thewaiting queue and/or to detect the direction of movement of gaps betweenthe objects. In this case, the propagation of the intermediate spaces tothe end or destination of the waiting queue is examined, in particular.Alternatively or additionally, the flow of movement, in particular as anoptical flow, can also be used as additional information in themonitoring scene.

In another possible embodiment of the invention, the object detectormodule is designed to detect an orientation of the objects as additionalinformation. This additional information can also be used by theevaluation device and/or the modeling module to identify and/or verifythe queue. In the case of people for example, it is possible to carryout facial recognition and/or skin/skin color recognition and in thismanner derive the orientation of the people. The queue shape can bedetected or modeled more easily from the orientation of the people sincepeople very often look in the direction of the end, that is to say thedestination or exit, of the queue. In the case of automobiles, it ispossible, for example, to detect the radiator, the license plate and/orthe windshield and to detect an orientation in this manner. Queuedetection is especially facilitated in the case of automobiles, inparticular, since they cannot turn around etc. in the waiting queue,like people, and the significance of the orientation is thus higher.

Another addition to the invention may provide for the evaluation deviceto have an environment module which is designed to accept and/oridentify a basic level of the monitoring scene in the at least onemonitoring area as additional information. As a result of the detectedbasic level, the localization of the objects can be improved andconsequently the queue can be identified and/or verified in a simplermanner.

The possibility of a depth chart module which provides a depth chart asadditional information relating to the monitoring scene or themonitoring area can also be considered to be an optional improvement,the evaluation device being designed to use the additional informationto identify and/or verify the queue. For example, such a depth chart canbe constructed from a stereo camera system or from other specialsensors.

Another possibility for improving the significance involves themonitoring image or a further monitoring image being in the form of athermal image, with the result that heat information, for examplehead/face/hands in the case of people and/or hood/exhaust in the case ofautomobiles, can be easily identified and this additional informationcan likewise be used to identify and/or verify the queue—for example bydetermining the orientation of the objects.

The invention also relates to a method and a computer program having thefeatures, among others, of claims 11 and 13, respectively.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features, advantages and effects of the invention emerge fromthe following description of preferred exemplary embodiments of theinvention and the accompanying figures, in which:

FIG. 1 shows a schematic block diagram of an apparatus for identifying awaiting queue as a first exemplary embodiment of the invention;

FIGS. 2 a-c show a monitoring image for illustrating the methodaccording to the invention;

FIGS. 3 a-d show a highly schematic illustration of waiting queues indifferent situations for illustrating the method.

DETAILED DESCRIPTION

FIG. 1 shows a highly schematic illustration of an apparatus 1 foridentifying a queue of objects in a monitoring area. The apparatus 1 maybe in the form of a personal computer or other digital processing unitand is used, in particular, to identify waiting queues of people.

FIG. 2 a shows, for example, a waiting queue 2 of individual people in aplan view from above in a monitoring image 3. The apparatus 1 makes itpossible to identify the waiting queue 2, as visualized in FIG. 2 b bythe solid line 4 through the waiting queue 2.

The apparatus 1 may have an assessment module 12 which, afteridentifying the waiting queue 2, provides characteristic variables ofthe waiting queue 2, for example the length of the queue, the number ofpeople waiting, the average waiting time, the profile of the queue, thenumber of objects/people in the waiting queue 2, etc.

The apparatus 1 comprises an evaluation device 5 and an interface 6 viawhich the evaluation device 5 is or can be connected to an image source7. The image source 7 may be, for example, in the form of a monitoringcamera which is directed toward the monitoring area. In particular, amonitoring camera, as is often already installed in supermarkets, etc.,can be used as the image source 7, with the result that here noadditional hardware costs for the monitoring camera arise whenintegrating the waiting queue identification system. Alternatively, theimage source 7 may also be in the form of an image memory which providesstored monitoring images 3 which show the waiting queue 2 in amonitoring area.

After at least one of the monitoring images 3 has been passed to theevaluation device 5, the monitoring image 3 is examined by an objectdetector module 8 for search objects 10, for people in the case of FIG.2. For this purpose, the object detector module has one or moredetectors 9 which carry out content-sensitive object detection withinthe monitoring image 3. In particular, the detectors 9 are designed insuch a manner that the search for the objects 10 can be carried outwithout a background subtraction, that is to say in the monitoring image3 with a scene background (the square in this case). The detectors 9look for the objects 10, for example by using a multiplicity ofcomparison features which are evaluated using classification rules. Suchcontent-sensitive object detectors are known in image processing.

After some, a plurality of or all objects 10 have been identified in themonitoring image 3, the information relating to the objects 10 is passedto a modeling module 11. The modeling module 11 is designed to merge theexistence and the distribution of the objects 10 in the monitoring image3 with models of waiting queues, with the result that a waiting queue ismodeled on the basis of the identification of the objects 10. In thiscase, the prepared models are designed for all queue shapes: straight,curved, branched, with gaps, with agglomerations, etc. The models may bepredefined as analytical models, for example in the form of functions,or else as shapes.

After the waiting queue has been identified and/or modeled in themodeling module 11, the data relating to the waiting queue 2 areforwarded to the result module 12 which analyzes the waiting queue 2with respect to the number of people waiting, the waiting time, etc. andoutputs these characteristic variables of the waiting queue 2.

The described apparatus 1 can be used, for example, in supermarkets, atticket counters, etc. in order to open further checkouts or counters inan automated manner if a predefined queue length or waiting duration isexceeded. It is also possible for the marketing department to useinformation relating to the detected queue shape to place products oradvertisements in the waiting areas which have been determined.

In the basic form illustrated, it is possible to already identify awaiting queue 2 by analyzing a single monitoring image 3.

Adding further functional modules makes it possible to improve thedetection accuracy of the apparatus 1. These further functional modulesare additional options which do not appear to be absolutely necessary:

A movement analysis module 13 is designed to determine movementinformation from the monitoring image 3 or from at least two monitoringimages 3 as an item of additional information:

For example, movement patterns of the waiting queue 2 can be identifiedby analyzing the movement of the objects 10 themselves or by analyzingthe optical flow in the monitoring image 3. It is possible to detect themovement of the objects 10 in the queue 2, as schematically indicated inFIG. 2 c by means of arrows 14. Alternatively or additionally, theintermediate spaces between the objects 10 may be observed and theirpropagation to the end of the waiting queue 2 can be analyzed. Thedirection of movement of the objects 10, of the intermediate spaces orof the optical flow results in a clear indication of the shape of thewaiting queue 2.

An environment module 15 is designed to identify a basic level of theobserved scene or to acquire it by means of a user input. Knowledge ofthe basic level of the observed scene makes it possible to assign a 3Dmodel to the latter, with the result that the positions of the objects10 can be defined more accurately. This basic level is also an item ofadditional information.

The precise position of the objects 10 can be obtained as a further itemof additional information in a depth chart module 16 which uses, as aninput, the images from a stereo camera or special sensors, for example.

Further possibilities for obtaining additional information are the useof infrared cameras in order to identify, for example, heat informationrelating to the head, face and hands in the case of people or thehood/exhaust in the case of automobiles as objects. This heatinformation can be used to improve the detection of the objects 10 or todetect an orientation of the objects 10. This information is alsoadditional information.

Said additional information is used individually or together or in anydesired selection in the modeling module 11 in order to improve themodeling of the waiting queue 2. In order to improve the modeling, it isadditionally or alternatively also possible for a user to explicitlystate the shape or destination of a waiting queue to be expected.

FIGS. 3 a-d illustrate different situations of a waiting queue 2 inorder to illustrate a selection of different versions of the waitingqueue 2. According to FIG. 3 a, people as objects 10 may conglomerate ina waiting queue 2. According to FIG. 3 b, gaps may occur between thepeople as objects 10. These gaps may also be used to identify thedirection of movement of the waiting queue 2. Furthermore, as shown inFIG. 3 c, divisions or branches of the objects 10 within the queue 2 mayoccur.

FIG. 3 d illustrates the possibility of the evaluation of theorientation of the people 10 in the queue 2 and their direction ofmovement, illustrated by arrows, being able to help with theidentification of the waiting queue 2. The direction of movement isdetermined, for example, by comparing a monitoring image 3 at a time z=t and a monitoring image 3 at a time z=t+x, in particular by trackingan object. The determination can be carried out, for example, by themovement analysis module 13. The orientation of the objects 10 can bedirectly detected using special sensors, for example thermal imagingcameras. Alternatively or additionally, the orientation can also beeffected by the object detector module 8. The direction of movementand/or orientation of the objects 10 can be used as additionalinformation in the modeling module 11 for identifying the waiting queue2.

1. An apparatus for identifying a queue of objects in a monitoring area,the apparatus having an interface which can be connected to an imagesource, the interface being designed to accept at least one monitoringimage of the monitoring area from the image source, the monitoring imageshowing a scene background of the monitoring area with possible objectsand an evaluation device which is designed to identify the queue ofobjects in the at least one monitoring image, the evaluation devicehaving an object detector module which is designed to detect amultiplicity of the objects on the basis of the monitoring image, themultiplicity of the detected objects forming the basis for identifyingthe queue of objects, wherein the object detector module is designed toidentify the objects in the monitoring image with the scene background,the object detector module has content-sensitive detectors for detectingthe objects, or both.
 2. The apparatus as claimed in claim 1, whereinthe content-sensitive detectors comprise a multiplicity of comparisonfeatures and classification rules for the comparison features.
 3. Theapparatus as claimed in claim 1, wherein the evaluation device has amodeling module which is designed to model the queue on the basis of thedetected objects.
 4. The apparatus as claimed in claim 3, wherein themodeling module is designed to place a predefined waiting queue modelonto the multiplicity of objects.
 5. The apparatus as claimed in claim1, wherein the evaluation device is designed to identify the queue onthe basis of a single monitoring image.
 6. The apparatus as claimed inclaim 1, wherein the evaluation device has a movement analysis modulewhich is designed to detect the direction of movement of the detectedobjects and/or the direction of movement of gaps between the objects,and/or the flow of movement in the monitoring image as additionalinformation on the basis of at least two monitoring images, or acombination thereof, the evaluation device, being designed to use theadditional information to identify the queue.
 7. The apparatus asclaimed in claim 1, wherein the object detector module is designed todetect an orientation of the objects as additional information, theevaluation device being designed to use the additional information toidentify the queue.
 8. The apparatus as claimed in claim 1, wherein theevaluation device has an environment module which is designed toidentify a basic level of the monitoring image in the at least onemonitoring area as additional information, the evaluation device beingdesigned to use the additional information to identify the queue.
 9. Theapparatus as claimed in claim 1, wherein the evaluation device has adepth chart module which is designed to identify a depth chart asadditional information, the evaluation device being designed to use theadditional information to identify the queue.
 10. The apparatus asclaimed in claim 1, wherein the at least one monitoring image or afurther monitoring image is in the form of a thermal image.
 11. A methodfor identifying a queue of objects in a monitoring area, at least onemonitoring image of the monitoring area being passed from an imagesource to an evaluation device, the monitoring image showing a scenebackground of the monitoring area with possible objects, the evaluationdevice identifying the queue of objects in the at least one monitoringimage, an object detector module of the evaluation device detecting amultiplicity of the objects on the basis of the monitoring image, themultiplicity of the detected objects forming the basis for identifyingthe queue of objects, wherein the object detector module identifies theobjects in the monitoring image with the scene background, the objectdetector module identifies the objects using content-sensitivedetectors, or both.
 12. The method as claimed in claim 11, characterizedby the use of the apparatus as claimed in claim
 1. 13. A computerprogram having program code means for carrying out all of the steps ofthe method as claimed in claim 11 when the program is executed on acomputer.